GOFAI meets generative AI: Development of expert systems by means of large language models

E.C. Garrido-Merchán, C. Puente

Knowledge-Based Systems Vol. 351, nº. Part A, pp. 116657

Summary:

The development of large language models (LLMs) has transformed knowledge-based systems such as open-domain question answering, which can automatically produce vast amounts of seemingly coherent information. Yet these models suffer from well-known limitations, most notably hallucinations, the confident generation of incorrect or unverifiable facts. In this paper, we introduce an approach to the development of expert systems that uses LLMs in a controlled and transparent way. By restricting the domain and employing a structured, prompt-based extraction protocol, we distil the knowledge of an LLM into a symbolic Prolog representation that can be inspected, validated and corrected by human experts, and queried by a deterministic inference engine. We evaluate the approach along three complementary axes. First, we assess factual accuracy through a manual expert audit and, to address the limited scale of manual checking, a large-scale automated cross-validation against Wikidata over three different LLM families (Claude Sonnet 3.7, GPT-4.1 and Grok 3), reporting Wilson confidence intervals and explicitly separating entity-linking error from factual error by means of a type-aware entity linker. Second, we show that the generated knowledge bases support genuine multi-hop deductive reasoning, negation as failure and aggregation when executed in SWI-Prolog, confirming that they behave as functioning expert systems rather than inert fact lists. Third, we analyze the statistical guarantees of the pipeline, making explicit how the autoregressive, non-independent nature of LLM generation affects the concentration bounds and confidence intervals. The result is a transparent hybrid solution that combines the recall of LLMs with the precision and explainability of symbolic systems, laying the foundation for dependable AI applications in sensitive domains. All code, prompts and generated knowledge bases will be made publicly available at https://github.com/eduardogarrido90/GOFAIGenAI upon acceptance.

Spanish layman's summary:

Los grandes modelos de lenguaje pueden inventar datos. Este trabajo los usa de forma controlada para extraer conocimiento y convertirlo en reglas Prolog que un experto valida y corrige, generando sistemas expertos transparentes y fiables en dominios sensibles.

English layman's summary:

Large language models can invent facts. This work uses them in a controlled way to extract knowledge and turn it into Prolog rules that experts can validate and correct, yielding transparent and reliable expert systems for sensitive domains.

Keywords: Expert systems; Language models; LLMs; Fact-Checking; hallucinations

JCR-JIF Impact Factor and WoS quartile: 8,000 - Q1 (2025)

DOI reference: https://doi.org/10.1016/j.knosys.2026.116657

Published on paper: October 2026.

Published on-line: July 2026.

Citation:
E.C. Garrido-Merchán, C. Puente, "GOFAI meets generative AI: Development of expert systems by means of large language models", Knowledge-Based Systems, Vol. 351, nº. Part A, pp. 116657, October 2026. [Online: July 2026] doi: 10.1016/j.knosys.2026.116657